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ABSTRACT 

The Colorado Learner Needs Assessment (CLNA) is 
described, with operational models for sampling, computer analysis 
and reporting. The rationale of the program is discussed, followed by 
its specifications (scope, reporting possibilities, precision and 
data format) , development (meetings, letters, preliminary tryout, 
sampling and analysis plans, field work plans, administration, 
scoring and analysis, and mailing results) , and uses. (KM) 
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FOREWORD 



This report contains technical information regarding materials and 
procedures for learner needs assessment as developed at the Colorado 
Department of Education. JFull. and constructive use of the needs 
assessment depends a great deal upon the sensitivity and understanding 
with which it is used. It is to promote such use that this report is 
intended. 

We welcome your comments and questions regarding this information. 
Through common understandings and cooperative endeavors, we in Colorado 
may increase our abilities to identify and understand the needs of the young 
learners in Colorado schools. 



Donald D. Woodington, 
Commissioner of Education 
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I. PURPOSES 

The Colorado Learner Needs Assessment (CLNA) provides data useful for 
developing authoritative statements regarding critical needs among learners 
statewide, in certain pupil population groups, and in those participating school 
districts where district results are requested. Critical learner needs, defined 
as "discrepancies between goals and their achievement 11 , guide the development 
and funding of innovative and exemplary programs, as provided u&der the Elemen- 
tary and Secondary Education Act, Title III. 

At the same time, the CLNA demonstrates methods for assessment which may 
be adapted for use in local school districts and othe*. states. Operational 
models for sampling, computer analysis, and reporting, along with supporting 
rationale comprise the CLNA, as described in the remainder of this report. 

II. .RATIONALE 

The Colorado Learner Needs Assessment (CLNA) is based on certain assump- 
tions regarding: (1) the nature of learner needs, (2) criterion and norm-referenced 
assessment, (3) measurement and evaluation, and (4) multiple uses of a single assess- 
ment. 

The Nature of Learner Needs 

Certain commonly-held views on what children need to learn in school are 
stated in goals such as this one, which is one of several adopted by the Colo- 
rado State Board of Education in 1971: 

Each student in the state has the opportunity to acquire THE TECHNIQUES 
OF LEARNING which make discovery of knowledge and wisdom a functional* 
exciting and lifelong process. 

The CLNA provides a means to determine learner needs in relation to general, 
long-range educational goals such as the one above. Doing so requires certain 
assumptions to be made regarding definition of "learner needs 11 , relationships 
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of individual and group needs, how to identify critical learner needs, a distinc- 
tion between learner and program needs, and the importance of learner readiness 
in determining learner needs. 

"Learner needs " defined . Central to the CLNA is the definition of "learner 
needs" as "the discrepancy between goals and their achievement." Basic to this 
definition is the assumption that particular skills, attitudes and knowledge which 
enable progress toward long-range goals can be identified by professional judgment, 
based on what is known about how children learn. Once broad, long-range goals 
are restated into more specific objectives, appropriate measures can be developed. 

Individual and group needs . The CLNA was designed to yield information 
of the educational needs of groups of learners rather than individual students. 
This is not intended in any way to minimize the importance cf needs of individual 
students; rather, the intent is to provide general indicators of group needs to 
guide district program development, not individual student information for planning 
specific lessons. 

Critical needs . Learner needs may be considered to be critical wi hey 
seriously jeopardize prospects for achievement of long-range educational goals. 
Criticality increases with (a) the importance of a particular skill, knowledge 
or attitude in achieving educational goals; (b) the relative numbers or per- 
centages of students lacking such important skills, knowledge, and attitudes; and 
(c) the concentration of many needs among certain students. Whether a need is 
critical calls for a judgment based on cooperative deliberation over results 
from the CLNA and other relevant data. Such deliberations often result in a 
better understanding of learner needs by the persons involved. 

Learner needs and program needs . A clear distinction between learner 
needs and urogram needs is necessary for conducting an accurate assessment of 
learner needs and for effectively allocating resources to meet these respective 
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needs. Once learner needs can be accurately and comprehensively assessed, 
program needs in terms of materials, teacher capabilities and other resources 
can be specified. The teacher who tells her class, "This test is to help me 
see how I'm doing as much as it is to help you see how you are doing," illustrates 
one constructive view of the relationship between learner needs and program needs; 
she is willing to modify her program on the basis of assessed learner needs. 

Readiness . The CLNA was developed according to generally expected or 
average levels of educational development. The user must consider the 
appropriateness of the CLNA for a particular group of students before stating 
conclusively that those students need to learn specific things found to be 
lacking by the CLNA. A review of the group's past learning experiences nnd 
achievements will provide an indication of a general readiness level, which 
should be considered in developing statements ^f learner needs. 
Criterion and Norm-referenced Assessment 

To determine the extent to which certain goals and performance objectives 
are being achieved, many different types of assessment may be used. The CLNA 
utilizes two of the more common approaches to assessment--criterion-referenced 
and norm-referenced. 

Norm-referenced measures are constructed to distinguish between good 
spellers and poor spellers, students with large and small vocabularies, with 
strong and weak mathematical skills, with positive and negative attitudes, and 
so on. Exercises comprising norm-referenced instruments are selected on the 
basis of their power to make such distinctions. Instruments made up of such 
exercises are administered to a sample group (i.e., students in grade 3) and 
from the obtained score distribution are derived standardized scales, such as 
stanines and percentiles, by which scores from subsequent testing may be 
compared. 



Criterion-referenced tests, on the other hand, are made up of exercises 
which indicate in specific detail whether students can or cannot perform certain 
tasks. An exercise or group of exercises, for example, may be developed to 
indicate whether or not students can add two three-digit numbers, with carrying. 
Instead of being chosen for their power to differentiate among skillful and non- 
skillful students as for norm-referenced tests, exercises for criterion-referenced 
tests are chosen to elicit skills, knowledge or attitudes as specified in commonly- 
desired objectives. 

Results from both norm and criterion-referenced tests may be interpreted 
in relation to goals and objectives sought in the school program. Norm-referenced 
results may be most useful in summarising learner needs in terms of general gosls 
whiJ criterion-referenced interpretations may be most useful for more detailed 
diagnosis of learner needs in terms of specific performance objectives. The CLNA 
was developed to yield both norm and criterion-referenced data. 
Measurement and Evaluation 

Two different but related operations comprise the needs assessment process 
of the CLNA — measurement and evaluation. Generally, '.he Colorado Department of 
Education provides the CLNA as a means for measurement ; and local districts evaluate 
their own results in terms of local goals and objectives. State- local cooperation 
is essential to constructive applications of the CLNA. 

Measurement . Obtaining reliable and relevant data on learner needs requires 
specifying the skills, knowledge and attitudes to be measured, selecting appropriate 
means of measurement, gathering data according to standard procedures, estimating 
the precision of the data gathered, and setting out the results in a simple and 
understandable form. Emphasis of the measurement operation is on the technical 
adequacy of the procedures, adherence thereto, and the estimation of probable 
measurement error. Precision is sought in measurement activities. 
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Evaluation . Once the data have been gathered and analyzed, attention 
turns to their interpretation and constructive uses. This calls for professional 
judgment regarding (1) the adequacy of the measures in terms of relevance and 
precision, and (2) the nature and cricicality of learner needs indicated in the 
data. Generally, the rule to follow is this: If important objectives are being 
measured, if students show weaknesses in demonstrating such learning, and if the 
measures are 01 sufficient precision, the user may then conclude that learner 
needs are indeed present and may proceed to describe these needs in the detail 
required to guide remedial ef'^rts. Judgment, informed by research and tempered 
b> experience, is sought in evaluation activities. 
Multiple Uses 

The CLNA was designed to yield data useful at both the state and local 
district levels of educational responsibility. Specific uses depend on the 
informational requirements at hand along with the skill and interest of the 
user > to relate the results to commonly-sought goals and performance objectives. 

State use . State use of results from the CLNA is in relation to 
"Educational Goals for Colorado Citizens," providing statewide indicators of 
learner needs in relation to these commonly-sought goals. Resources mae'e 
available under the Elementary and Secondary Education Act, Title III have been 
directed toward developing programs to meet critical needs thus identified. 

Local use . The data provided each participating district may also be used 
for identifying learner needs and planning remediation. Also, procedures of 
sampling, instrument development and data analysis developed at the Colorado 
Department of Education may be adapted for use in local ~~honl districts, to 
supplement the data gathered by the CLNA and local testing programs* 
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III. SPECIFICATIONS 

Consistent with the purposes and rationale as described previously, 
specifications are described here regarding administration, scope, reporting 
possibilities, precision of the CLNA and data format. 
Administration 

— ~~~~ — — t 
Specifications regarding administration of the CLNA are as follows: 

Student time: 1 hour for skills assessment 

20 minutes for attitude assessment 
No more than 300 students per 
district per grade 

CDE responsibilities: Offer to each district superintendent 

Mailing assessment materials 
Scoring and computer analysis 
Mailing results to district superintendent 
Providing suggestions for use of results 

District responsibilities: Requesting to participate 

Distributing materials to schools 
Administering instruments 
Returning materials to CDE 
Regulating consultant help 
Reporting irregularities 

For districts with more than 200 students per grade level, sampling 
instructions are provided for each participating school. This entails select- 
ing students via systematic random sampling procedures from a grade-level 
listing. In the largest district, schools to participate are selected on a 
stratified random procedure at the Department of Education based on informa- 
tion provided by the district. 
Scope 

The CLNA provides district-wide indicators of student needs in three 
common curricular areas and general attitude toward school. There is no 
intent to assess the extent to which all goals and objectives are being achieved. 
Rather, the intent is to provide performance information relating to some of the 
goals and objectives commonly sought across the state. 
O 6 

ERLC 



Reading . Students demonstrate their skills of reading comprehension 

and vocabulary via multiple-choice exercises which call for skills of: 

Recognizing inferences and implications of written material 
Distinguishing fact from opinion 

Recognizing the purposes of characters portrayed in stories 
Identifying principal ideas and topic sentences 
Putting sequential events in order 

Identifying words most different in meaning from given words 

Language arts * The CLNA assesses student skills in four common areas of 

English language arts: 

Spelling — students identify one misspelled word among others spelled 
correctly 

Usage (or grammar) — students identify wor I > and phrases correctly 

completing a sentence or paragraph or identify one incorrect sentence 

among two which are grammatically correct 
Capitalization — students distinguish among words which are properly 

capitalized and those which are not 
Punctuation — students distinguish between correct and incorrect usage 

of commas, colons* quotation marks and other commonly used marks 

of punctuation 

Mathematics . Students demonstrate their understanding of arithmatic 

concepts by selecting correct answers to exercises involving: 

Currency, decimals and fractions 
Equations 

Time, rate and distance 
Numerals and number systems 

Graphical representation of quantitative relationships 
Attitude toward school . A general, district-wide indication of attitude 
toward school among students at the secondary school level is obtained through 
a two-page questionnaire including semantic-differential and agree-disagree 
response modes. Students are asked for their candid opinions (which are to be 
kept anonymous) regarding various aspects of their school experiences. 
Reporting Possibilities 

Results, in computer-printout form, for each participating school district 
are sent to the respective Superintendent. State data and reporting suggestions 
accompany each district report. Three separate tables of data are provided. 
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Item -level performance . From percentages of students answering each skill 
exercise correctly in the district and statewide, such reports as this may be 
developed locally: 

Figure 1. 
Example E>*ercise Report 

"On this exercise, 66% of the students in our district at Grade 5 
answered this exercise correctly while 85% did so statewide: 

The greatest number, using the digits 3, 9, 7, 6 only once is: 

j a) 3679 b) 9376 c) 7963 *d) 9763 3) 6739 

District = 66% 
State = 85% 

"Based on this and other information already on hand we are con- 
sidering ways to improve our student's understanding of place 

value in our elementary mathematics program." j 

i 

. _ .« 

As with all data obtained by sampling, the user needs to know of probable 
error in the results due to sampling. This is given as a confidence interval within 
which the "true" district-state discrepancy probably lies. (See Figure 6, page 17) 

Average percent correct . The average percent of the exercises in each 
skill area which were answered correctly summarizes the item-level performance as 
described previously. Within-district comparisons may be made using the percentile 
and stanine scales provided by the CLNA as follows: 

(a) The district percentile, indicating the percentage of districts 
scoring below the district reported, and 

(b) The district stanine which is a normal transformation of the 
percentile scale, compensating for the tendency of district scores 
to cluster at the state average. 

Within-district comparisons may be shown graphically by such graphs as 

these taken from the computer printout shown in Figure 5, page 16 of this report. 
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Figure. 2. 
Example of a District Summary 



D 

I 

S 

T 

R 

I 

C 

T 



S 
T 
A 
N 
I 
N 
E 
S 



9 

8 
7 
6 
5 

3 
2 
1 



a 

3 

J 



DO 

c 

•H 

•o 

CO 
0) 
OS 



Grade 5 



4J 



0) 






•o 


60 


00 


3 


CO 


c 


4J 


3 


•H 


•H 


00 


■o 


4J 


c 


CO 




CO 




< 


-3 





CO 



Grade 11 



Such comparisons across grade levels among the skill areas may be made 
with the percentiles and stanines, but not the average percent correct. Relative 
difficulty of the exercises affects the average percent correct statistic but not 
percentiles and stanines, which may be considered as common scales permitting 
within district comparisons. Also, the user should keep in mind that the school 
attitude score is a scale score , not a "percentage correct", with the higher 
scale score indicating a more positive attitude toward school. 

District "expected" scores . Data are provided each participating district 
regarding the discrepancy between an "expected" score, based on certain economic 
factors, and the score actually obtained xn a district. Such a graph as that 
shown on the next page may be developed from this data given in Figure 6, page 16 
of this report. 
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Figure 3. 

Example of a District Comparison of Obtained and "Expected" Scores 
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Precision 

Studies of the precision of the CLNA show it to be of more than sufficient 
validity and reliability to indicate learner needs in each participating district 
and statewide. 

Validity . The CLNA is valid to the extent it taps skills and attitudes 
commonly sought in a district's educational program. Accordingly, validation 
procedures started with the development of the instruments, when exercises were 
discarded, accepted, or revised on the basis of the professional judgments of 
curriculum experts, teachers, md the developers of the instruments. Further 
validation of the CLNA was obtained in a questionnaire survey in districts 
participating in the assessment of 1971. 
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Judgments of relevance varied from district to district, but indicated a 
general agreement that the measures covered some (but certainly not all) of the 
objectives sought in their educational programs. 

From the preliminary tryout of the Attitude-Toward-School Inventory, a 
factor analytic study was conducted to find student response patterns. Eight 
factors were tentatively identified as contributing to students' general attitudes 
toward school: 

1. Attitude toward school worth. 

2. Attitude toward teachers. 

3. Attitude toward school relevance. 

4. Attitude toward avoidance of school. 

5. Undefined. 

6. Attitude toward dropping out. 

7. Undefined. 

8. Attitude toward length of school experience. 

Findings from this study are very tentative and, for further validation, require 
further study of data and experience gained from use of the Inventory statewide. 
Validity of the Attitude Inventory rests partially on the assumption that the 
students, unidentified in the scoring of the Inventory, will give candid responses 
on their attitude toward common aspects of their school experience. 

Reliability . District averages; obtained on the CLNA are sufficiently 
reliable that the user can be confident » within certain limits, that the same 
averages would be obtained on repeated measurements. A single measurement 
may be considered a sample of a large number of measurements which could be 
taken and which would probably yield slightly different results. The average 
of these repeated measurements may be considered to be a "true" scor<?. Informa- 
tion regarding the reliability of the CLNA is set out in T<?.ble I. 
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Table I 

Errors of Measurement for District Averages in Skills Assessment 



Grade 


Test 


No. of 
Districts 


Reliability 
Coefficients 


Index of 
Reliability 


Standard error 
of Measurement 







» 








5 


| Lang. Arts 


104 


.93 


.97 


2.5% 


5 


Reading 


104 


.95 


.98 


2.1% 


5 


Math 


104 


.90 


.95 


3.3% 


11 


i 

j Lang. Arts 


99 


.86 


-93 


3.9% 


11 


! Reading 


99 


.86 


.93 


4.1% 


11 


Math 

i 


99 


.92 


.96 


3.0% 



The information in Table I shows that the reliability of the district 
averages in each of three achievement areas at both grade levels is very high. 
For example, on grade five Language Arts tests the index of reliability was 
approximately .97. This indicates that the obtained district means would 
correlate about .97 with "true*' district means, that is, means that would 
result from perfectly reliable tests — tests containing no random measurement 
error. The standard error of measurement, is the standard deviation of the 
differences between obtained and "true" district averages. For example, if 
in district X the "true" average of all examinees on the fifth-grade Language 
Arts test was 74 percent, the standard error of measurement indicates how close 
to the mean of 74 percent that the district average would be expected to be. 
For the grade fivr Language Arts test, the value of 2.5% indicates that about 
two-thirds of the districts had averages that did not differ by more than 2.5% 
from their "true" averages. These small values for the standard errors of 
measurement show that the obtained district averages differ little from their 
"true" averages. 

From a study conducted by the Laboratory of Educational Research at the 
University of Colorado the following data were reported regarding the amount 
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of district-to-district variation around the state average. 



Table II. 

State Averages, Standard Deviations of District Averages, 
and Range of District Averages for Each Test (corrected for chance) 



Grade 


State* Average 


Std. Dev. of 
Dist. Ave. 


Range of 
Dist. Ave. 


5 


Lang. Arts 


57% 


8% 


36%- 75% 


5 


Reading 


55% 


9% 


27%- 76% 


5 


Math 


41% 


5% 


15%-63% 


11 


Lang. Arts 


65% 


7% 


47%-84% 


11 


Reading 


47% 


5% 


13%-68% 


11 


Math 


56% 


10% 


24%- 78% 



*State average was defined as the average of the means from the 112 



participating Colorado school districts. 

The standard deviation values range from 5% to 10% and indicate that 
there are considerable differences among school districts in the levels of 
pupil achievement. For example > in Grade 11 math, the standard deviation of 
10% indicates that approximately the highest-scoring one-sixth of the districts 
had averages of 66% or greater, whereas the lowest-scoring districts had averages 
of 46% or less. Such district-to-district variation permits development of 
measuring scales which reflect real differences in performance rather than 
sampling and measurement error. 
Data Format 

The data are reported to each participating district to enable both criterion 
and norm-referenced interpretation. Usually a coml ination of these two approaches 
is necessary to give a complete and understandable view of learner needs within a 
district and statewide* 

Exercise analysis . As shown in Figure 4, following, statistics for each 
skill exercise are provided for the participating district as compared with 
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statewide statistics. To interpret Figure 4, the user of the CLNA needs a copy 
of the skill exercise to determine what particular skill is assessed. Users should 
keep in mind that these exercises were chosen both to represent commonly-sought 
educational objectives and to distinguish between generally skillful and non- 
skillful learners. For example, a few exercises may indicate capabilities of 
capitalization among students in the district;a single exercise may indicate 
student ability to capitalize the first word of a direct quotation. 

District averages . Figures 5 and 6 show the printouts for the average 
percentages of exercises answered correctly in each skill area for the participating 
district and statewide. On the student Attitude Inventory, a district scale score 
indicates the extent to which students have generally positive attitudes toward 
their school environments. 

Stanines and percentiles are provided to enable local users of the CLNA to 
make within-district comparisons among the educational levels and curricular areas 
represented. Possibilities for reporting learner needs in relation to local goals 
and objectives were illustrated previously in this report on pages 8 to 10 . 
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That is > \812)of thedistrict's students who were tested answered item 1 
correctlyV^wereas^852)of the students in tie statewide sample answered item I 
correctly. .In othervords, (?D (81-85) fewer of the district's students answered 
correctly tkrn for th^ statewide sample. Considering the s«mpiing enror, 
tjje "true" percent for. the district is probably souewher* beiveen 0y and 
(^31 that is xhe discrepancy could have resulted for chance qs shown by 

tKevfact thatXthe confidence includes ^vhe value "0" f hence thr district figure 
is nobsreliablv different from the sijti figure, 
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Figure 4. Sample Printout 
District Performance on Skill Exercise 
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At grade 5, (Si) students were tested; at grade 11, (jO$) students were tested. 
In the district, the average (aeaa) score for theL-uViuage Arts test war jIZ^ 
for the statewide saa*>le, the average score was incorrect. The distr^t 
wan on the libguage arts test exceeded (MJ/of t*se for other participating 
districts. Con sidering sa mpling error, ti\e true/mean of the district would 
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Figure 5 . Sample Printout 
District Averages 
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The average score of tested 5th grade students in tills district was(£l£) 
on the Language Arts tests, whereas the average score in similagd iytric-s 
was<j8|) The discrepancy of Q7p (582-A12) was su^h that only(2%^of the 
districts had a poorer showing^ this test. The confidence iMerval 



^pgn to -l3*H indicates that the discrepancy cannot be atttiMited to sampling 
error asSSated with the sample *size. The district v s performance at grade 
litprdly poor, but at gradp 11 the p*tern was tnudi better. The stud 

'/rikng(94Slof similar 



>erun 

5 was unl^prdly poor, but at gradf 11 the p*tern was tnudr be ttej . The student 
de I 

\ 
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\ 



« 

* 

* 

« 
« 
« 
« 

« 
« 
* 

« 
« 
« 
« 
« 
« 
« 
« 
« 
•ft 
ft 



\ N \ / I 

DlSTpXrT o£HPU«SANCe iV- DELATION To EXPfCTEj A^PORMaNCt. 

iT"""" 



TEST 



AVE»AG\ «-t»<C€MTyOr * . 



ACTUAL \**ECTE0 O^TAIN>0 C • I N J 1 • PtjuCjMlLE ST jMJNp 



\ 



\ 



DISTRICT ulaC 



GfUOE « 

LANGUAGE ARTS 
BEAD IMG 
MATHFMATICS 

GRAOE u 

LANGUAGE mHTS 

READING 
MATHEMATICS 



38 

?8 



<3> 

57 
A3 



<3> 



-n 

-t's 




i 3 



64 

53 
64 



66 
4V 
49 



4 

5 



■5 1 
0* 
1« 



1 
4 



34 



1 46 

73 



I 



4 



avEKAGt 0IST«1CT uISCHe^CT 

ACTUAL EAP£CTE0 n3TAlN;E0 Cr)NF.X^T t PEpC^MIlE S^NlNe 
3«A0E * I 

SCHOOL ATUTUOE S3 *8 5 3t f © a 



n 
* 

ft 
ft 

ft 

41 
ft 
ft 
ft 
ft 
0 
ft 
0 



ft 
# 
ft 
ft 
ft 
ft 
• ft 

ft 
ft 
ft 

ft 

ft 
ft 
ft 
ft 




Figure 6 , Sample Printout 
District Expectancies 
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Table III 



Stanlnes and Percentiles Corresponding to District Averages 
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The stanine scale Is a standard score scale that avoid* some of 
the faults of the bore easily understood percentile scale. Percentile 
units near the median are "bunched up," hence the performance differ- 
ence between the 40th and 50th percentiles is much less than the 
difference between the 80th and 90th percentiles. The stanine units 
are equal (except for stanines 1 and 9) hence if stanines are used 
there is less danger of over interpreting large differences in percen- 
tiles near 50 and underinterpreting small differences in precentiles 
near the extremes. The figure below illustrates the stanine scale. 

Figure 7 , 

The Stanine Scale in Relation to the Normal Distribution, 
Illustrating the Percents o r Districts in Various Stanine 

Categories 




Stanine i 2 3 4 5 6.7 8 9 



Table IV. 

Comparison cf Stanines and Percentile Ranges 
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IV. DEVELOPMENT 

Developing the measures was a cooperative venture, involving personnel 
from school districts, the Colorado Department of Education, and the Laboratory 
of Educational Research at the University of Colorado. The major events in the 
developmental process are indicated in the diagram below and are summarized 
following. 



Figure 8 Flow Chart of Major 
Activities in Developing Colorado's Learner Needs Assessment 
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The Colorado Department of Education had chief responsibilitiy for the 
following activities: 1, 2, 3, 4, 6, and 14. 

Colorado University's Laboratory of Educational Research had chief 
responsibility for the following activities: 5, 7, 8, 9, 10, 11, and 13. 

Personnel in Colorado school districts had chief responsibility for 
activitiy 12. 

1. Initial Decision 

The decision to develop new measures was made after an analysis of the 
data already on hand across the state. The data on hand was found to be too 
dx verse and fragmented to use as common measures of learner needs. 

2. Meetings with District Personnel 

Concerns expressed in meetings with curriculum and research directors from 
various districts were that: 

1. The projram be voluntary 

2. Unfair comparisons among districts be avoided 

3. The diversity of curricular emphases among districts be considered 

4. New data be related to that already on hand. 

All of these concerns were observed in the development of the measures. 

3. Contract with University of Colorado 

The Department entered into a contract with the Laboratory of Educational 
Research to develop measures with sufficient precision to yield district 
averages in three pupil skill areas and pupil attitudes toward school. A prior 
contract was renegotiated to include an exercise-by-exercise analysis of the 
skill measures* 

4. Letter of Invitation to District Superintendents 

The Department offered the measures to District Superintendents on a 
voluntary basis explaining the nature and purpose of* the measures. 
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5. Developing Exercise Pools 

The staff of the Laboratory of Educational Research inspected: 

• Curricular materials commonly used in Colorado schools 

• Objectives judged to be important by Colorado teachers 

• Commonly-used achievement tests to develop pools of 

exercises calling for skills described in these sources 

Approximately 120 exercises were developed or collected from the various 
sources for each program area at each level* At the same time, a prelim- 
inary form of the attitude inventory was constructed. 

6. Listing of Participating Districts 

This was completed approximately 2 weeks after the offer was made to 
District Superintendents. 

7. Preliminary Tryouts 

Tryout of exercises in the pools was conducted by the Laboratory of 
Educational Research in the Boulder, Colorado schools. An effort was made 
to select schools typifying all those in Colorado in t*nns of student chara- 
cteristics. Statistical analyses of the data securec from approximately 125 
students for each test included: 

Basic skills - (a) item intercorrelations to identify items measuring 

similar skills 

(b) point biserial correlation to identify exercises 
which generally distinguish between skilled and 
non-skilled students (see Table VIII) 

(c) item difficulty in terms of percentages of students 
answering correctly. 

Attitude inventory - Factor analysis to identify response patterns among 

students regarding certain aspects of their school 
experiences. 

In addition to these statistical data, judgments were obtained from 
classroom teachers regarding the relevance of each exercise to common school 
curricula. 
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The following topical outline was constructed to guide construction of 
the final forms. 



{ Grades 5 and 11 | 



| Grade 8 j 



Language Arts 



Reading 



Attitude toward School 



Spelling Vocabulary 
Capitalization Comprehension 
Punctuation 
Grammar 



Length of school experience 

Teachers 

Learning 

School worth 

School relevance 

Avoidance of school 



8. Sampling and Analysis Plans 

To gain indicators ' f learner needs for a district, samples were drawn to 
represent aJl students reaching certain levels in certain curricular areas as 
follows: 



Selection of students representative of all those served by the program 
was accomplished by school personnel with instructions provided by the Laboratory 
of Educational Research. Selection of exercises generally representative of goals 
and objectives commonly sought in the curricular areas listed above was 
accomplished by procedures described in step No. 7 as described above. 

Concurrently with the sampling plans, the Laboratory of Educational 
Research planned the computer analysis of results to be consistent with the 
legal authority and assumptions described in Section II. 



Educational Level 



Curricular Area 



Grade 5 
Grade 6 
Grade 11 



Reading, Language Skills, Mathematical Skills 
Attitude Toward School 

Reading, Language Skills, Mathematical Skills 
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9. Final Forms 

Based on the procedures described in No. 7 above, exercises were selected 
to make up internally-consistent instruments to measure the skills and attitudes 
among students in district programs, according to specifications described in 
Section III above. Instruments were delivered in camera-ready form by the 
Laboratory to the Department in time for printing and distribution to districts 
in December, 1971. 

10. Field Work Plan s 

Field work plans, including test-accounting, monitoring and further 
communication with participating districts, v°re worked out upon completion 
of the sampling plans. 

11 . Explanatory Materials 

Materials explaining the measurement results as well as suggested activities 
for district? were prepared concurrently with the scoring and computer analysis 
of the measurement data. 

12. Administration of Instruments 

School district personnel received materials by mail, administered the 
instruments according to iristructions and returned the completed forms and 
answer sheets to the Department either by mail or by hand. 

13. Scoring and Analysis 

Scoring and computer analysis were accomplished at the University of 
Colorado computer center. Scoring utilized a correction for guessing. 

14. Mailing Results to Districts 

Each participating Superintendent obtained results for his district 
before January 1, 1972. 



24 



Table V. 

Instructional Areas Assessed: Grade 5 



Language Arts Elements 


Exercise Numbers 


No. of Exercises 


Spelling 
Capitalization 
Punctuation 
Usage (Grammar) 
Alphabetizing 


1 through 5 
6 through 19 
20 through 26 
27 through 39 
40 


5 
14 

7 
13 

_i 

Total 40 


Reading Elements 


Exercise Numbers 


No. of Exercises 


Vocabulary 
Comprehension 


41 through 57, 

72 through 80 
58 through 71 


26 
14 

Total 40 


Mathematics Elements 


Exercise Numbers 


No. of Exercises 


Computation 
Concepts 

Application 

Interpreting Graphical 
Representations 


81, 90, 92 

82, 87, 89, 91, 96, 
97, 99, 100 

84, 88, 94, 95, 98 

83, 85, 86, 93 


3 

8 
5 

_4 

Total 20 


Grand Total = 

Testing Time = 1 hour (20 minutes 
per area measured) 


■ 100 Exercises 

i 
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Table VI. 



Instruction Areas Assessed: Grade 11 



Language Arts Elements 


Exercise Numbers 


No, 


of Exercises 


Spelling 
Capitalization 
Punctuation 
Usage (Grammar) 


1 through 8 
9 through 21 
22 through 26 
27 through 40 


Total 


8 
13 
5 

14 
40 


Reading Elements 


Exercise Nunbers 


No. 


of Exercises 


Vocabulary 

Paragraph Comprehension 


41 through j>3 
54 through 70 


Total 


13 
17 
30 


Mathematics Elements 


Exercise Numbers 


No. 


of Exercises 


Application 

Interpreting Graphical 
Representation 


71 through 78, 
81 through 90 

79, 80 


Total 


18 

2 
20 


» « 

Grand Total 

Testing Time * 1 hour (20 minutes 
per area measured) 


* 90 Exercises 



In considering the scope of the CLNA t users must keep in mind that broad 
program areas and total population of students are represented in the sampling. 
The results are intended to be general indicators for district-level use and 
not specific measures for school or classroom use. 
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Typical Exercises 
Grade 11 Math: 

Mny were i» *« ch b0 * 7 

.)2A b)36 c)«0 d)576 

far did ht got 

«) 78 miles *> 189 * tlM 

Grade 11 Reading: 



Thre.teacd inetitutions . like endengered epeci... h.v. •'^J?°£"* cad 
remerkeble power, of .urviv.l. There U the Rew> Chureh, the Brltieh Houee 
oHord.. the Ceraen Gen.r.1 St.ff end the Hie. Amricn P.geent. Criticised 
u UlyUhite.... de.on.treted .g.inst by Wo«n'e Lib, condescended to by 
"tell.ctuel. eao the New York Ti.ee (which, he. beeaJ»ovn to epere two peregrephe 
leep in Sd. to report the wiw^KU. Wc« «n«.lly bloon. like • crop 
S !.« e««er corn. The eecond S.turdey night in Septe*.r 
then 60 Billion televiewere tuning in ee, live fro. AtUntic City, Bert Park, 
open, the le.t envelope, milk, the l«t drop of euepenoe, nnnouncee the winner 
end leunchee the pageent'e there eong: There She Ie. 

—Time Mgsziue 

Which oi the following is not true? 

a) lha author of the parsgraph feels ttat the New York Tinea gives generous 
coverage to the Miss America Pegesnt. 

b) ThrSuSor feels th.t the Mis. Africa Psge ant is a threatened institution 

c) The author considers the pageant to be over-dramatic. 

Whst does the author mean when he says "Miss America annually blooms like 
a crop of late simmer corn?" 

a) The Miss America Pageant is corny. 

b) The pageant always take piece in late susner. M 

c) Miss America has most often been selected from the corn states. 

What is meant by the term "lily-white" as used in the psragraph? 

a) The pageant is a very clean affair. 

b) There sre always a lot of white-lilies at the psgeant. 

c) Pew Blacks or other minority persons psrtlcipste. 
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Grade 11 Language Arts: 



Indicate the word or words which correctly coaplett tht sentence. 

th€ * ***** t0 tHo gaae, they did not so to tht d*nc« afttrwarde. 

a) While 

b) Although 



Ihe shaded portions of the dlagran below represent what pert of the flfure? 



e) 1/2 b) 5/10 c) 3/9 d) 5/12 e) 4/9 

Jane's bike is i 1/3 yard* long. How ueny feet long Is her bike? 
a) 4 ft. b) 5 ft. c) 12 ft. d) 13 ft. e) 1X3 ft. " 

John bought lunch 17 days In Sept eater. Each lunch coat 40 cents, how auch 
did all the lunches coat? 

a) $4.80 b) $5.80 c) $6.80 d) $6.90 a) e.SOc 



She awea 



t t but her golf fane was poor. 



a) good 

b) well 



The captain 



hardly see the lighthouse. 



Grade 5 Math: 




Graile 5 Language Arts: 




the one word which correctly completes the 



sentence when it appears In 



The children 



down on the grass* 



a) aet 

b) aat 

c) eetted 
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TABLE VII. 



Item-Analysis Data 
On Basic Skills Instruments 

Grade 5 



Item 
No. 


P- 


r. 


Item 
No. 




r . 


Item 
No. 


P. 


r. 


* 

1 


.89 


. 34 


1 C 

35 


.51 


.38 


69 


.64 


.05 


2 


.86 


. 32 


36 


.61 


.42 


70 


.73 


.23 


3 


0£ 

.86 


.22 


37 


• 74 


/ A 

.49 




.38 


.20 


* 


OA 

.89 


OO 

. 38 


*5 O 

38 


.75 


• 46 


72 


.68 


.27 


-> 


A A 

.90 


• 36 


OA 

39 


CO 

.58 


• 53 


73 


.48 


.35 


o 


• OO 


. 3/ 


/.A 

40 


co 
. 52 


A 1 

.41 


74 


"T A 

.70 


/ O 

.48 


7 


*? A 

. /9 


CO 

. 52 


A 1 

41 


oc 
.85 


• 37 


75 


• 63 


• 37 


8 


0£ 

.86 


• 42 


4Z 


"7 A 

.79 


. 33 


76 


«56 


.42 


Q 


o o 
.83 


• 4Z 


43 


.48 


o c 
• 35 


7 "T 

11 


7A 

. 70 


A 1 

.41 


1 A 

10 


•7 A 

.74 


A 7 

.47 


A A 

44 


.61 


OA 

.29 


"TO 

78 


CO 

.58 


.31 


11 


o o 
. 38 


.07 


A C 

45 


.80 


^ A 

.29 


79 


• 60 


.27 


1* 




1 A 

.39 


A£ 
40 


. Bo 


1 A 

.3* 


OA 

80 


. / I 


A "T 

.47 


13 


.81 


• 18 


A 7 

4/ 


.63 


.20 


O 1 

81 


.73 


.26 


1 A 

14 


.82 


. 43 


AO 

48 


. 72 


.35 


82 


.74 


o2 


15 


c c 


. 40 


A A 
H? 


. 79 


o o 


o o 

83 


.48 


^3 


lo 


.75 


. 52 


CA 

5U 


. 77 


o o 
.32 


84 


. 76 


*• c 

. :5 


17 


.85 


.46 


51 


.58 


.21 


85 


.45 


-32 


18 


.68 


.28 


52 


.54 


.32 


86 


.67 


.23 


19 


.79 


.53 


53 


.64 


.40 


87 


.38 


.21 


20 


.72 


.41 


54 


.77 


.35 


88 


.53 




21 


.85 


.37 


55 


.76 


.38 


89 


.35 




22 


.73 


.38 


56 


.43 


.14 


90 


.61 


'."C 


23 


.50 


.21 


57 


.49 


.30 


91 


.58 


7 


24 


.34 


.13 


58 


.78 


.35 


92 


.58 


->5 


25 


.59 


.42 


59 


.56 


.24 


93 


.40 


.17 


26 


.45 


.35 


60 


.56 


.27 


94 


.5j 


.It 


27 


.76 


.47 


61 


.88 


.34 


95 


.34 


.2i 


28 


.60 


.49 


62 


.77 


.22 


96 


.24 


.18 


29 


.63 


.28 


63 


.79 


.21 


97 


.46 


.21 


30 


.50 


.34 


64 


.77 


.32 


98 


.26 


.11 


31 


.48 


.37 


65 


.86 


.35 


99 


.41 


.12 


32 


.52 


.38 


66 


.80 


.36 


100 


.27 


.27 


33 


.59 


.43 


67 


.67 


.31 








34 


.77 


.46 


68 


.61 


.20 









p * proportion of a sample (N*500) answering item correctly 
r « the item-total point-biserial correlation 



EMC 
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TABLE VIII. 



Item •Analysis Data 
On Basic Skills Instruments 



Grade 11 



Na 
WO . 


.. ?' 


r . 


item 

NO. 


-P- . 


r. 


Item 
No. 


. P- 


r. 


1 


• 70 


. 65 


! 31 


.81 


.30 


61 


.76 


.40 


L 


HQ 
. Oo 


.67 


i 32 


.89 


.42 


62 


.33 


.30 


1 
J 


• 00 


. 23 


33 


.62 


.13 


63 


.56 


.51 




e i 




34 


.67 


.47 


64 


.73 


.06 


c 


.01 


.63 


35 


.68 


.38 


65 


.69 


.56 


0 


• Jo 


. 36 


36 


.60 


.58 


66 


.67 


.30 


7 




• 01 


37 


.46 


AO 

.08 


! 67 


.66 


.54 


8 


.55 


.57 


38 


.56 


.08 


68 


.33 


.25 


9 


.73 


.09 


39 


.45 


.39 


i 69 


.20 


.19 


10 


.83 


.56 


40 


.38 


.43 


70 


.43 


.42 


11 


.81 


.44 


41 


.50 


.19 


i 71 


.79 


.60 


12 


.77 


.69 


42 


.58 


.49 


72 


.82 


.45 


13 


.84 


.69 


43 


.84 


.31 


73 


.84 


.51 


14 


.81 


.14 


44 


.55 


.49 


74 


.70 


.65 


15 


.87 


.52 


45 


.79 


.27 


7 5 


.75 


.60 


16 


.88 


.34 


46 


.82 


.48 


76 


.61 


.62 


17 


.81 


.73 


47 


.78 


.44 


77 


.69 


.57 


18 


.90 


.40 


48 


.87 


.42 


78 


.69 


.33 


19 


.82 


.68 


49 


.68 


.36 


79 


.72 


.41 


20 


.77 


.57 


50 


.54 


.49 


80 


.68 


.53 


21 


.80 


.21 


51 


.51 


.48 


81 


.63 


.51 


22 


.73 


.16 


52 


.77 


.41 


82 


.46 


.60 


23 


.69 


.58 


53 


.56 


.39 


83 


.67 


.44 


24 


.70 


.53 


54 


.48 


.07 


84 


.54 


.59 


25 


.83 


.39 


55 


.30 


.07 


85 


.36 


.56 


26 


.76 


.71 


56 


.61 


.49 


86 


.46 


.62 


27 


.46 


.49 


57 


.47 


.16 


87 


.48 


.22 


28 


.67 


.21 


58 


.59 


.46 


88 


.32 


.53 


29 


.76 


.40 


59 


.72 


.38 


89 


.29 


.28 


30 


.91 


.42 


60 


.75 


.53 


90 


.37 


.49 



? » proportion of a sample (N-500) ar /ering item correctly 
r « item-total po in t-bi serial correlation 
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V. Uses 

One Example 

Perhaps the most constructive use of the CLNA can be made in school 
districts where people look at their programs for explanations of why their 
results are high, low or average. In one school district, for example, the 
Superintendent found the averages for his district to be considerably below 
the state averages. His staff told him that perhaps students who could be 
classified as educationally handicapped had participated and brought the 
district average down. 

The Superintendent's first inclination was to re-compute the district 
results, excluding the answer sheets of certain students. On further 
deliberation, however, the Superintendent and his staff decided to look at 
the district's program for appropriateness for all of the students within the 
district. Figure 9., following, illustrates this use of the CLNA for 
program improvement. 



OPERATIONS 

Selection of materials, 
staffing, instructional 
procedures. 



INPUTS 

Goals and Objectives, readiness, 
t district characteristics, 
| funding, etc. 



OUTPUTS 

Measures of skills, knowledge, 
attitudes, judgments of program 
effectiveness, etc. 



Figure 9. 
A System for Relating Measures of 
Learner Needs to other program Characteristics* 
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The district Superintendent and his associates whose deliberations were 
just described illustrate in principle the model shown in Figure 9. First, 
they used the results of the measurement as "Output 1, Indicators for their 
district's program* Second, looking for explanations for the relatively low 
results, the staff considered "Inputs" in terms of abilities of the students 
involved* Their final decision, however, was to consider modifications in the 
"Operations," especially in providing for students who may not be benefiting 
fully from the present program. (Subsequent measures may reveal gains in the 
total district program performance due to increased attention to particular 
needs among certain groups of students*) 

Once learner needs can be traced to deficiencies in current educational 
program practices, then allocation of resources to bolster or modify weak 
programs is a relatively simple operation* Applications for ESEA, Title III 
funds, for example, are approved on the basis of whether real learner needs 
have been identified and whether the proposed program activities are likely to 
ameliorate these needs* 

An Overview 

To determine who used what results from the assessment of 1971, and for 
what purposes, the Department coordinated a questionnaire survey among school 
personnel who participated in the Fall 1971 assessment* Findings from the 
survey are given in * special report available from the Department and are 
summarized below* 

Who used the results? 

The results appear to be of more use to administrators and teachers 
than to accountability committees, school boards, and other staff* Adminis- 
trators in 31 districts made "much use" of the testing results. School 
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boards in 17 districts made "little use" of the result^. The large number 
(14) of "not applicable" responses for other staff may be due to the 
absence of staff other than teachers and administrators in many of the 
smaller participating districts. 

Persons from two districts mentioned that results were .lso useful 
to counselors and one person mentioned "parents" as a group to whom the 
results were useful. Three different comments indicated an intent to use 
the results as time and resources permitted; lack of clerical help in 
tabulating the responses and the press of other responsibilities were 
given as reasons for not using the results. 

Uhich materials were of most use? 

The average percent correct appeared to be che most useful statistic 
to local district personnel, followed closely by the percent of students 
answering each exercise correctly. The confidence interval, perhaps 
because of its complexity* appeared to be somewhat less useful to cistricc 
personnel. 

The explanatory materials and worksheets received average ratings, 
indicating considerable usefulness for a majority of districts participating 
in the voluntary measurement project. 

Several respondents requested more detailed analysis, specifically; 
(1) an item analysis on the attitude survey, (2) results for each subtest 
(vocabulary, spelling, etc.), (3) school by school analysis when there 
was more than one school per district administering a given measure, 
(4) a report of raw scores, and (5) national as well as state 
comparisons. 
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Were results relevant to local informational needs ? 

Perhaps the most important of the questions posed in the survey 
pertained to the relevance of the measures to local informational needs. 

Judgments as to the relevance of the measures to local goals and 
objectives were distributed a]] jng the little-much" rating scale, 
with some tendency toward "much relevance". Approximately 25 percent of the 
responses (20 of 81) judged "much relevance" of the measures in identifying 
learner needs. 

General Conclusions 

1. Because many of the ratings were above average on the utility of the 
measures, and because at least half of the districts scored below 
the state average, it would appear that the measures were considered 
to be useful even in some districts scoring relatively low on the 
measures. 

2. The explanatory materials, worksheets, and instructions to the test 
coordinators were judged to be useful to a rather high degree in a 
majority of districts. The actual use of these materials, as indicated 
in the comments, varied a great deal from considerable use in some 
districts to more in other districts. 

3. Results of the measures appeared to be most useful to administrators 
and teachers, with administrators using the results more than teachers. 
This is not surprising as the results were reported for entire districts 
rather than by school, classroom or individual students. 

4. Relevance of the assessment to local goals and performance objectives 
varied somewhat from district to d let, but was generally rated high. 

5. Many districts lack the adequate resources to make full and constructive 
use of the results from the CT.NA. 
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